Sparse basis selection, ICA, and majorization: towards a unified perspective

Sparse solutions to the linear inverse problem Ax = y and the determination of an environmentally adapted overcomplete dictionary (the columns of A) depend upon the choice of a " regulariz-ing function " dx in several recently proposed procedures. We discuss the interpretation of dx within a Bayesian framework, and the desirable properties that " good " (i.e., sparsity ensuring) regularizing functions, dx might have. These properties are: Schur-concavity (dx is consistent with majorization); concav-ity (dx has sparse minima); parameterizability (dx is drawn from a large, parameterizable class); and factorizability of the gradient of dx in a certain manner. The last property (which naturally leads one to consider separable regularizing functions) allows dx to be efficiently minimized subject to Ax = y using an Affine Scaling Transformation (AST)-like algorithm " adapted " to the choice of dx. A Bayesian framework allows the algorithm to be interpreted as an Independent Component Analysis (ICA) procedure .